A Meta-Study of Software Visualization Effectiveness

The overriding goal of software visualization (SV) technology is to be effective—that is, to provide functionality that people actually want to use (usefulness), and to provide non-problematic access to that functionality (usability). Nonetheless, the eight extant taxonomies of SV focus almost exclusively on technology expressiveness: the kinds of visualizations that can be produced with a system, as well as the methods prescribed by the system for generating and interacting with those visualizations. However, while expressiveness is necessary for effectiveness, it certainly is not sufficient. To enhance the navigational markings of the expressiveness-laden map charted by past SV taxonomies, this meta-study critically assesses the state-of-the-art with respect to SV effectiveness. The meta-study’s focus on effectiveness implies the need to abandon both the SV system as its unit of analysis, and the SV systems literature as its principal data. Accordingly, SV is reconceptualized as a collection of human tasks; the empirical studies that have considered those tasks form the study’s data. The meta-study’s data differ along three key dimensions: their theories of effectiveness, their research techniques, and their research foci. Different choices along any of these dimensions lead to accounts of effectiveness that differ markedly—both qualitatively and quantitatively. To draw out the differences to which alternative choices lead, the meta-study pursues in-depth analyses of divisions of labor (indicating theories of effectiveness); units of analysis, data collection/analysis methods, and desired results (indicating research techniques); and research questions, SV artifacts, target programs, participants, and SV tasks (research foci). In light of the SV effectiveness studies’ overriding interest in pertaining to SV in practice, a provisional analysis of their ecological validity is additionally offered. A synthesis of the analysis both proposes theories of effectiveness, research techniques, and research foci that appear ripe for future research, and recommends that future research be firmly grounded in ethnographic descriptions of SV in practice.